Co-regularization Based Analysis of Feature Sharing Algorithms

A common approach in domain adaptation (DA) [1] and multitask learning (MTL) [2] is to create an expanded feature representation by sharing features across domains (in DA) or across tasks (in MTL) and then learning a classifier over this expanded feature set. In this paper, we refer to such techniques as feature sharing algorithms (FSA). One such FSA is EASYADAPT [1], which takes each feature in the original problem and replicates it three times: general, source-specific and target-specific. In this extended feature space the source data will contain general and sourcespecific features whereas the target data will contain general and target-specific features. EASYADAPT is simple, easy to implement as a preprocessing step and outperforms many existing techniques [1, 3], namely, SOURCEONLY (target hypothesis trained on labeled source data only), TARGETONLY (target hypothesis trained on labeled target data only), ALL (combination of source and target labeled data) and PRIOR (target hypothesis trained with SOURCEONLY as a prior on the weight vector) [4]. However, a theoretical analysis of why EASYADAPT performs better than the other aforementioned approaches is clearly missing. In this abstract, we present such an analysis. In order to achieve our goal, we model EASYADAPT in terms of co-regularization. This is an idea that originated in the context of multiview learning and for which there exists some theoretical analysis [5]. We denote source and target empirical errors for some hypothesis h as ǫ̂s(h) and ǫ̂t(h) and the corresponding expected errors as ǫs(h) and ǫt(h). PRIOR and EASYADAPT optimize the following cost functions: